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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Program  Executive  Officer,  SHIPS 
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•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 
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Abstract 

DoD  energy  inefficiency  is  a  significant  liability  and  a  constraint  on  operations  and  a  force- 
protection  challenge.  It  is  therefore  imperative  to  reduce  energy  demand  and  provide 
operational  forces  greater  flexibility  among  alternative  energy  sources.  However,  the  current 
acquisition  processes  undervalue  technologies  with  the  potential  to  improve  energy 
efficiency.  We  report  the  results  of  leveraging  an  innovative  platform,  the  Massive  Multiplayer 
Online  Wargame  Leveraging  the  Internet  (MMOWGLI)  to  link  and  elicit  collective  intelligence 
from  the  acquisition  community  for  the  challenge  of  DoD  energy  inefficiency.  We  first  linked 
the  existing  MMOWGLI  energy  data  with  samples  of  acquisition  data  using  lexical  link 
analysis  (LLA).  We  generated  match  matrices  based  on  themes  discovered  in  both  data  sets. 
The  themes  and  match  matrices  helped  identify  the  gaps  and  opportunities  to  apply  collective 
intelligence  from  the  MMOWGLI  game  to  the  current  acquisition  process.  This  effort 
demonstrates  superb  potential  of  an  innovative  methodology  that  can  be  deployed  quickly  to 
mobilize  the  intellectual  capacities  of  the  acquisition  community.  It  may  also  increase  the 
overall  awareness  of  ongoing  acquisition  research  to  warfighters  and  create  a  positive  impact 
for  the  future  acquisition  decisions  to  help  achieve  improved  DoD  energy  efficiency. 

Background,  Needs,  and  Research  Questions 

Studies  evaluating  the  DoD’s  energy  use  have  been  conducted  by  the  Institute  for 
Defense  Analyses,  the  Defense  Science  Board  Energy  Security  Task  Force,  and  JASON 
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(an  independent  scientific  advisory  group).  All  three  studies  suggest  that  DoD  energy 
inefficiency  is  a  significant  liability,  a  constraint  on  operations,  and  a  force-protection 
challenge.  More  specifically,  all  three  studies  led  to  two  consistent  requirements  for  DoD 
energy  efficiency:  (1)  By  reducing  energy  demand,  we  may  provide  operational  forces 
greater  flexibility  and  reduce  their  dependency  on  logistics  infrastructure;  and  (2)  We  can 
improve  the  DoD’s  current  requirements  and  acquisition  processes  to  value  the  technologies 
with  the  potential  to  improve  energy  efficiency  (DoD  Acquisition  and  Technology,  2012). 

The  Massive  Multiplayer  Online  Wargame  Leveraging  the  Internet  (MMOWGLI), 
sponsored  by  the  Office  of  Naval  Research  (ONR),  is  an  online  game  platform  designed  to 
elicit  collective  intelligence  from  an  engaged  pool  of  world-wide  players.  The  Naval 
Postgraduate  School  (NPS)  is  one  of  the  primary  developers  of  the  game  software. 

Recently,  the  Navy’s  Energy  and  Environmental  Readiness  Division  (OPNAV  N45),  hosted 
by  NPS  Modeling  Virtual  Environments  and  Simulation  (MOVES)  Institute,  conducted  a  civic 
and  military  collaboration  specifically  for  examining  Navy  energy  efficiency  May  22-25.  In 
the  past,  the  NPS  hosted  a  series  of  successful  games,  piracyMMOWGLI  (201 1-present, 
ongoing)  and  energyMMOWGLI  (May  2012),  which  built  the  critical  mass  of  players  needed 
to  find  creative  solutions  to  the  real-life  difficult  problems,  such  as  piracy  and  energy. 

In  the  energyMMOWGLI  game,  ideas  were  collected  through  “play  an  idea  card”  and 
“take  action,”  as  shown  in  Figure  1.  The  motivating  “call  to  action”  for  players  is  to  improve 
the  U.S.  Navy’s  combat  capability  and  energy  security,  particularly  by  promoting  energy 
efficiency,  reducing  energy  consumption,  and  diversifying  its  energy  supply  (use  of 
alternative  energy)  for  the  sake  of  future  strategic  readiness.  The  overall  goal  is  to  reduce 
reliance  on  fossil  fuels  from  overseas. 


^pggmmo/  gli^^ 
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Figure  1.  The  energyMMOWGLI  Game 


In  this  energyMMOWGLI  game,  560  players  contributed  over  5000  ideas  and  68 
action  plans.  Lexical  link  analysis  (LLA;  Zhao,  Gallup,  &  MacKinnon,  2010,  2011a,  2011b, 
2011c,  2012)  was  used  in  analyzing  the  collected  data.  All  results  are  published  online  (see 
MMOWGLI  Energy  Game,  2012;  MMOWGLI  Energy  Game  Portal,  2012;  MMOWGLI 
Business  Initiative  [Bll]  Game,  2013;  MMOWGLI  Bll  Game  Portal,  2013). 

•  https://portal.mmowgli.nps.edu 

•  https://portal.mmowgli.nps.edu/energy-welcome 

•  http://web.mmowgli.nps.edu/energy/ldeaCardChainEnergy2012.html 

•  http://web.mmowgli.nps.edu/energy/ActionPlanListEnergy2012.html 

We  leveraged  the  energyMMOWGLI  game  in  the  acquisition  community  through  the 
following  four-step  process.  Further  details  appear  later  in  this  paper  and  in  the  online  game 
portal. 
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1 .  Prepare  acquisition  data.  Collate  key  terms  and  goal  statements  of  current 
acquisition  programs  within  the  congressional  budget  processes  for  use  by 
the  LLA  methodology. 

2.  Perform  link  analysis  and  correlation.  Compare  the  already-collected 
energyMMOWGLI  results  to  determine  action  plan  relevance  on  a  program- 
by-program  basis. 

3.  Design  new  capabilities  for  information  collection.  Define  questions  for  a 
continuation  round  of  the  energyMMOWGLI  game,  to  support  programmatic 
life-cycle  needs  of  the  acquisition  community. 

4.  Plan/conduct  follow-on  games.  Conduct  a  follow-on  game  focused  on  shared 
needs  of  many  energy  programs,  demonstrating  the  value  of  this  approach  in 
a  formal,  repeatable  way. 

Methodology 
MMOWGLI  Game 

The  game  is  built  using  a  unique,  open  source,  software  adaptation  of  the  Institute 
for  the  Future  (IFTF) — designed  game  to  simulate  a  real  world  “brainstorm.”  A  player  needs 
to  register  with  a  required  game  identification  (ID)  and  e-mail.  First  and  last  name  and  other 
personal  identification  information  (Pll)  are  not  required. 

The  game  starts  with  the  explanation  of  the  situation  and  allows  a  player  to  “play  an 
idea”  or  “take  action.”  Users  can  then  choose  to  input  an  idea  or  participate  in  the  discussion 
of  an  existing  idea  in  the  categories  of  “Innovate”  and  “Defend.”  The  discussion  can  be  in 
one  of  the  following  categories:  expand — build  on  this  idea  to  amply  the  impact;  counter — 
challenge  this  idea;  adapt — take  this  idea  in  a  different  direction;  explore — something 
missing?;  or  ask  a  question,  as  shown  in  Figure  2. 

In  the  end,  the  system  will  gather  collective  intelligence  that  resides  in  color-coded, 
tree-structured  sets  of  ideas  and  discussions  in  text  format  as  shown  in  Figure  3.  If  an  idea 
and  its  associated  discussion  have  merit,  which  is  determined  in  the  combination  of  the 
player’s  score  and  the  Game  Master’s  recommendation,  it  will  be  taken  into  a  separate  “take 
action”  board  for  further  planning  and  deliberation. 


Figure  2.  Categories  of  Ideas  Based  on  the  Styles  of  Responses 
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Figure  3.  Ideas  Collected  in  the  Color-Coded  Tree-Structured  Categories 

The  MMOWGLI  platform  is  suitable  for  tackling  a  broad  range  of  challenges  for 
national  security,  multiple  stakeholders,  and  challenges  for  small  or  big  communities  (e.g., 
corporations  and  research  communities  like  the  acquisition  system  community).  It  is  a 
configurable  innovation  platform  that  can  be  adapted  to  any  scenario.  For  example,  an 
aerospace  and  defense  company,  Raytheon,  is  considering  the  game  engine  for  use  within 
a  company  as  a  corporate  innovation  platform. 

Lexical  Link  Analysis 

LLA  is  a  form  of  text  mining  in  which  word  meanings  represented  in  lexical  terms 
(e.g.,  word  pairs)  can  be  represented  as  if  they  are  in  a  community  of  a  word  network  (Zhao 
et  al.,  2010,  2011a,  2011b,  2011c,  2012).  LLA  “discovers”  and  displays  these  networks  of 
word  pairs  from  large-scale  unstructured  data.  It  can  be  installed  as  a  search  and 
knowledge  management  tool  for  scoring  and  ranking  interesting  information  and  for 
visualizing  and  reporting  correlations  among  categories  and  layers  of  information  including 
lexical,  semantic,  and  social  links.  This  effort  then  presents  the  decision-maker  with 
previously  unavailable  and  emerging  patterns  and  themes,  as  well  as  unprecedented  levels 
of  analysis,  thus  reducing  the  workload  and  overcoming  the  blind  spots  of  human  analysts 
and  with  potential  automation.  For  example,  for  the  recent  MMOWGLI  games  used  to 
develop  and  identify  new  ideas  about  stated  subject  matters,  LLA  was  leveraged  to  identify 
potentially  interesting  information  from  “idea  cards,”  link  them,  then  recommend  them  to  the 
matched  action  plans  for  Game  Masters. 

Figure  4  shows  the  game’s  content  and  attributes,  which  were  processed  into  the 
inputs  (i.e.,  meta_data.txt  and  a  directory  of  text  files  with  idea  card  contents  to  LLA). 
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Figure  4.  Idea  Cards  Transformed  to  LLA  Inputs  (e.g.,  a  Directory  With  Files  of  Content 

of  the  Cards  and  Attributes,  meta_data.txt) 

There  are  two  steps  used  in  LLA  to  discover  themes.  A  theme  is  a  cluster  of  related 
word  pairs: 

•  1st  Iteration  (Figure  5  (a)):  Compute  word  pair  clusters  using  Newman 
community  finding  algorithm — words  as  in  a  community  (Girvan  &  Newman, 
2002). 

•  2nd  Iteration  (Figure  5  (b)):  Select  lexical  terms  linked  to  the  most  central 
nodes,  for  example,  “fuel,  shipboard,  liquid.” 
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Figure  5.  Two  Steps  LLA  Iterations  to  Group  Word  Pairs  Into  Themes 


Research  Results 

As  shown  in  Figure  6,  in  Phase  I,  we  planned  to  demonstrate  the  feasibility  of  the 
social  media  energyMMOWGLI  game  as  an  innovation  platform  that  could  generate 
valuable  and  unexpected  contributions  and  solutions  towards  the  DoD  energy  efficiency 
through  the  acquisition  process  by  linking  the  current  acquisition  programs  with  the 
energyMMOWGLI  game  using  LLA.  We  achieved  this  objective  through  performing  the 
tasks. 
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Figure  6.  A  Glance  of  the  Proposal  Objective 


Task  1:  Prepare  Acquisition  Data 

The  goal  here  is  to  collate  key  terms  from  the  current  acquisition  program  in  the 
congressional  budget  process.  The  congressional  budget  process  documents  (e.g.,  program 
elements  [PEs]  from  http://www.dtic.mil/descriptivesum)  will  be  used  in  this  task.  This  source 
is  the  accurate  and  authoritative  high-level  artifacts  under  the  DoD  Research,  Development, 
Test,  and  Evaluation  (RDT&E).  We  had  analyzed  part  of  these  documents  in  the  past  (Zhao 
et  al.,  2010,  2011a,  2011c,  2012)  in  detail  using  the  LLA  method  jointly  with  other  measures 
such  as  cost,  schedule,  and  performance. 

Specifically,  we  collected  the  following  most  recent  (2013)  tri-service  PE  documents 
for  this  project: 

•  http://www.dtic.mil/descriptivesum/Y201 3_Navy.html 

•  http://www.dtic.mil/descriptivesum/Y201 3_AirForce.html 

•  http://www.dtic.mil/descriptivesum/Y201 3_Army.html 

Task  2:  Perform  Analysis  and  Correlation 

Compare  the  already  collected  energyMMOWGLI  results  to  determine  action  plan 
relevance  on  a  program-by-program  basis. 

We  linked  the  energyMMOWGLI  data,  specifically,  38  action  plans  with  the  PEs 
prepared  in  Task  1 ,  and  224  Navy  PEs  to  evaluate  the  current  Navy  programs  relevant  to 
the  game  data.  Figure  7  shows  that  the  process  resulted  in  a  relevance  and  correlation 
matrix  as  illustrated. 
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Phase  I  Project:  Program  Relevance/Correlation  Matrix  of  MMOWGL!  Game 
Plans  -  Comparison  of  Key  Terms  from  Roth  Sources 


energyMMOWGU  Action  Plans 
[May  2012  Gam*  Data:  existing  corpus) 

Funded  Programs  via  PEs 
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x 


•Use  Samp  le  learned  threshold  or  LLA  measures  of  strength  of  correlation 

Figure  7.  Phase  I  Relevance  Matrix 


Figure  8.  The  Overall  Match  Matrix  for  the  MMOWGLI  Energy  Game  Action  Plans  and 

Navy  2013  Program  Elements 


Figure  8  shows  sorted  Navy  PEs  that  match  the  MMOWGLI  game  data  based  on  a 
sorted  LLA  score.  The  top  five  most  relevant  PEs  are  listed  as  follows: 

•  PE  0603724N:  Navy  Energy  Program 

•  PE  0601 153N:  Defense  Research  Sciences 

•  PE  06021 23N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Combat  Services  Support 
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Clicking  on  the  online  link  for  the  top  one  leads  to  the  online  page  of  the  “Navy 
Energy  Program,”  which  is  an  overall  PE  specifically  focusing  on  Navy  energy  issues  as 
shown  in  Figure  9.  This  validates  that  the  LLA  extracted  the  relevant  keywords  from  the 
game  data. 


The  matrix  in  Figure  8  shows  a  holistic  picture  of  the  current  acquisition  programs  in 
connection  with  the  DoD  energy  inefficiency  situations,  efficiency  requirements,  and 
possible  innovative  solutions.  Directly  looking  into  the  match  matrix,  as  illustrated  in  Figure 
8,  can  be  overwhelming.  For  that,  we  applied  LLA  to  discover  the  themes  and  divide  a  single 
match  matrix  into  many  match  matrices  in  different  themes.  For  our  research,  a  theme  is  a 
network  or  community  of  word  pairs  that  are  related  to  each  other.  To  discover  themes,  we 
first  applied  LLA  to  compute  word  pair  clusters  using  Newman  community  finding 
algorithm — words  as  in  a  community  (Girvan  &  Newman,  2002).  There  we  select  lexical 
terms  linked  to  the  most  central  nodes.  For  example,  shown  in  Figure  1 1 ,  the  red  nodes  are 
the  most  central  nodes  “environmental,  ship,  and  effective.”  The  red  links  are  the  word  pairs 
shared  by  both  sources  PEs  and  MMOWGLI  game  action  plans;  the  yellow  links  are  the 
word  pairs  unique  to  the  game  data;  and  the  green  ones  are  those  unique  to  the  PEs. 
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Figure  10. 


Themes  Discovered  for  Navy  2013  Program  Elements  Documents  and 
energyMMOWGLI  Data,  Thresholded  and  Then  Sorted  According  to  the 
Overlapped  Word  Pairs  From  the  Two  Sources 
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Red  nodes:  the  most  central  nodes,  used  as  keywords  to  summarize  this  theme 
Red  Links:  shared  by  at  least  two  sources 

Green  Links:  from  Source  ARPM^mmowoluenergy 
Blue  links:  from  Source 
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Figure  11.  Theme  395(E)  Link-Strength  Visualizations:  “Environmental,  Ship,  &  Effective” 

A  separate  matrix  can  be  constructed  for  each  theme  for  the  word  pairs  that  belongs 
to  only  a  theme.  Figure  12,  the  correlation  matrix  for  Theme  395(E)  labeled  as 
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“environmental,  ship,  &  effective,”  which  has  the  highest  matched  word  pairs  in  Figure  12. 
The  matched  PEs  are  sorted  according  to  the  number  of  matched  action  plans.  For 
example,  the  top  matched  PE  is  “0603724N_PB_2013,”  titled  “Navy  Energy  Program,” 
indicating  that  there  is  a  current  Navy  program  dedicated  to  “energy.” 

We  used  this  matrix  to  determine  where  opportunities  reside  in  the  current  process  to 
include  energy-related  elements.  Also  shown  in  Figure  12,  two  concepts,  “energy  efficient” 
and  “ship  design,”  are  dominant  in  this  theme.  They  are  dominant  because  there  are  four  (4) 
and  two  (2)  out  of  38  action  plans  contain  word  pairs  “energy  efficient”  and  “ship  design,” 
respectively.  This  seems  to  suggest  that  “efficient  energy”  may  have  to  work  with  the 
concept  “ship  design.”  However,  among  the  12  PEs  that  mentions  “ship  design,”  only  one 
entry  mentions  “energy  efficient.”  This  indicates  that  there  is  a  gap,  or  a  DoD  energy 
inefficiency  area,  and  therefore  an  opportunity  to  emphasize  the  concept  “energy  efficient”  in 
all  the  PEs  related  to  the  concept  “ship  design.” 
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Figure  12.  Match  Matrix  for  Theme  395(E) 


Following  the  same  analysis,  Appendix  A  lists  more  gap  and  opportunity  areas 
discovered  by  LLA. 

In  the  near  future,  we  will  engage  the  students,  faculties,  and  a  wide  acquisition 
research  community  to  continue  the  discussion  of  the  DoD  energy  efficiency  and  possible 
solutions  through  series  of  planned  MMOWGLI  games  (MMOWGLI  Energy  Game  Portal, 
2012).  As  possible  acquisition  professionals  being  Game  Masters,  the  brainstorming  and 
discussions  can  be  steered  towards  more  specific  requirements,  for  example,  the  ones 
below: 


1 .  How  to  provide  operational  forces  greater  flexibility  and  reduce  their 
dependency  on  logistics  infrastructure. 

2.  How  to  change  the  DoD’s  current  requirements  and  acquisition  processes  so 
they  do  not  undervalue  technologies  with  the  potential  to  improve  energy 
efficiency. 

The  results  from  the  match  matrices  can  be  recommended  areas  for  the  seed 
questions  for  a  MMOWGLI  energy  game. 

Conclusions 

Multiple  useful  conclusions  of  broad  applicability  arise  from  this  work. 
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•  We  demonstrated  the  use  of  the  MMOWGLI  social  media  brainstorming 
platform  and  LLA  as  a  combined  collective  intelligence  platform  to  gather 
consensus  via  the  MMOWGLI  energy  game  and  match  data  using  LLA,  with 
the  current  existing  DoD  programs,  derived  from  Navy  2013  PEs  documents. 

•  We  identified  critical  variables,  elements,  concepts,  or  word  pairs  that  can  be 
linked  to  Navy  energy  efficiency  within  and  among  numerous  programs. 

•  We  used  match  matrices  for  each  individual  theme  found  through  LLA  to 
identify  energy-related  parameters  or  elements  as  word  pairs,  and  then  we 
used  these  word  pairs  to  further  identify  opportunities  in  the  current  process, 
(i.e.,  what  PEs  might  be  good  candidates  to  engage  the  energy-related  action 
plans  discussed  in  the  MMOWGLI  energy  game?). 

•  We  found  that  the  great  majority  of  Navy  programs  are  affected  by  (or  even 
critically  dependent  on)  energy  issues,  but  goals  and  even  terms  are  handled 
inconsistently. 

Therefore,  without  imposing  significant  operational  burdens  and  vulnerabilities, 
innovative  “energy  efficiency”  ideas  from  the  social  media  game  might  be  quickly  and 
naturally  implemented  into  the  current  processes  that  drive  force  structures,  combat 
operations,  logistics,  and  acquisition  decisions. 

The  resulting  capability,  the  automation  of  LLA  computations  and  an  analyst 
interface  for  report  generation,  demonstrate  MMOWGLI  together  with  LLA  as  an  important 
tool  throughout  the  longer  life  cycle  of  the  acquisition  process  for  incorporating  the  “fully 
burdened  cost  of  fuel”  into  acquisition  analyses. 

Recommendations  for  Future  Work 

Much  work  can  continue;  specifically,  we  see  excellent  potential  in  the  following: 

•  Crowd  sourcing  to  provide  meaningful  feedback  on  either  cross-cutting 
themes  (such  as  energy  reduction/efficiency)  or  specific  acquisition 
programs. 

o  For  example,  acquisition  experts  might  participate  in  the  Business 
Innovation  Initiative  (bii)  MMOWGLI  Game  Round  2  in  summer  2013 
to  gain  further  experience  in  relevant  crowd-sourcing  capabilities. 

•  Building  MMOWGLI  game  infrastructure  in  tandem  with  LLA  computational 
structure  to  reduce  manual  labor  and  maximize  analyst  flexibility  with  each 
round. 

•  Continuing  work  on  real  datasets  that  spurs  meaningful  (rather  than  toy  or 
contrived)  analysis,  and  producing  further  data  visualizations  tuned  to  support 
focused  analytic  queries  by  players  and  decision-makers. 

•  Maintaining  backwards  compatibility  among  games  to  enable  steady  growth 
via  the  available  corpus  and  products  each  year.  This  further  enables 
longitudinal  analysis  and  observability  of  trends  and  evolution  over  time. 

•  Stabilizing  the  data-model  design  of  LLA  computational  products,  which  may 
enable  future  visualization  improvements  to  be  directly  applied  to  past 
products. 

•  Speedier  production  of  LLA  products  that  can  influence  fast-react  game 
rounds  or  program  changes  as  they  proceed,  rather  than  after  the  event.  We 
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want  to  reduce  analysis  cycles  from  weeks  to  days,  and  even  to  hours, 
approaching  real  time. 

•  Program-support  brainstorming  and  collective  intelligence  experiments  that 
should  continue,  both  for  proposed  and  current  programs  of  record.  Games  + 
link  analysis,  connecting  the  record  of  “what  is  reported  being  done”  with 
“what  do  people  think,”  all  help  normalize  the  use  of  concept  terminology  and 
also  identify  unsuspected  applicability  of  new  breakthrough  capabilities. 

•  Overall  progress  and  process  improvements  that  may  now  be  measured  so 
that  causes  and  effects  of  improvements  in  acquisition  system  cost- 
effectiveness  and  responsiveness  are  documented. 

•  Navy  strategies  for  improving  energy  efficiency  needs  to  be  handled 
consistently  across  programs.  Terms  of  reference,  metrics,  and  opportunities 
all  need  to  be  addressed  consciously  and  consistently. 

•  Following  a  series  of  deliberate  experiments,  long-term  procedural 
improvements  to  the  formal  milestone  acquisition  process  can  be  considered. 
For  example: 

o  Are  program  terms  of  reference  consistent  with  DoD-wide  best 
practice? 

o  Are  all  applicable  energy  reduction  and  energy  efficiency  techniques 
identified? 

o  Routine  crowd  sourcing  as  due  diligence:  subject-matter  expert  and 
public  reviews  (as  appropriate)  to  accompany  milestone  decisions. 

o  Has  in-game  or  post-game  analysis  identified  synergies  among 
different  programs  that  deserve  further  investigation? 

•  Open  question:  How  can  these  tools  statistically  identify  discussions  that  are 
focused  on  concepts  in  novel  combinations?  In  other  words,  are  they  “on 
topic”  but  not  explicitly  addressed  by  the  reference  documents?  These  are 
the  discussions  where  significant  innovation  may  be  occurring. 

•  Improving  the  defense  acquisition  process  is  a  major  challenge  that  holds 
potentially  massive  payoffs.  Decision-milestone  preparations  can  benefit  from 
broader  review  and  judicious  cross-program  comparisons  that  discover 
possibilities  that  aren’t  already  recognized.  Future  rounds  of  the  Bll 
MMOWGLI  game  will  continue  investigating  how  crowd-sourcing  techniques 
might  best  be  applied  to  make  a  good  acquisition  process  even  better. 
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Appendix  A:  Gaps  and  Opportunity  Areas  to  Integrate  the  Innovative  Concepts  and 
Action  Plans  From  the  MMOWGLI  Energy  Game  Into  Current  Navy  Program  Elements 

“Fuel,”  as  an  independent  variable,  can  be  crucial  for  improving  DoD  energy 
efficiency.  For  example,  according  to  the  DoD  energy  inefficiency  report  (DoD  Acquisition 
Technology,  2012), 

The  current  process  either  does  not  consider  fuel,  or  considers  only  the 
commodity  price.  However,  moving  fuel  into  and  around  the  theater  of 
combat  imposes  significant  operational  burdens  and  vulnerabilities,  drives 
force  structure  toward  support  at  the  expense  of  combat  operations,  and 
increases  costs  for  delivery  and  logistics.  Neither  current  requirements  nor 
acquisition  processes  accurately  explore  tradeoff  opportunities  using  fuel  as 
an  independent  variable.  This  prevents  an  end-to-end  view  of  fuel  utilization 
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and  distorts  platform  design  choices,  consequently  preventing  DoD  from 
achieving  maximum  combat  benefit  for  its  logistics  effort. 

We  argue  that  by  matching  the  data  and  consensus  gathered  from  the  collective 
intelligence  platform  (e.g.,  MMOWGLI  energy  game  data  with  the  current  existing  DOD 
programs,  exemplified  in  the  Navy  2013  PEs  documents),  we  can  identify  critical  variables, 
elements,  concepts  or  word  pairs  that  are  linked  to  energy.  Therefore,  without  imposing 
significant  operational  burdens  and  vulnerabilities,  innovative  “energy  efficiency”  ideas  from 
the  game  might  be  naturally  implemented  into  the  current  processes  that  drives  force 
structures,  combat  operations,  delivery,  and  logistics. 

We  use  match  matrices  for  each  individual  theme  found  through  LLA  to  identify 
energy-related  parameters  or  elements  as  word  pairs,  and  then  we  use  these  word  pairs  to 
identify  the  opportunities  in  the  current  process  (i.e.,  what  PEs  might  be  good  candidates  to 
engage  the  energy-related  parameters/elements/concepts/word  pairs  discussed  in  the 
MMOWGLI  energy  game).  These  findings  are  listed  below. 
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The  match  matrix  for  Theme  430  suggests  that  PEs  mentioned  the  concepts 
“existing  fleet,”  “shipboard  system(s),”  “shipboard  equipment,”  and  “secondary  power”  that 
might  have  the  overall  potential  to  engage  Action  Plans  10,  26,  and  18. 

•  Action  Plan  1 0:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 

•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 

•  Action  Plan  18:  Offshore  basing. 
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The  match  matrix  for  Theme  393  suggests  that  the  PEs  with  the  concepts  “Navy 
energy,”  “energy  systems,”  “power  generation,”  “alternative  fuel,”  “alternative  energy,” 
“renewable  sources,”  and  “costs — energy/infrastructure”  could  be  used  good  candidates  to 
implement  the  innovative  ideas  related  to  Action  Plans  11,  1 8,  22,  and  35. 

•  Action  Plan  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet. 

•  Action  Plan  18:  Offshore  basing. 


•  Action  Plan  22:  Scaling  the  small  solutions:  Energy  recycling  and  rethinking 
“The  Big  Fix.” 
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The  match  matrix  for  Theme  458  shows  that  the  PEs  mentioned  (“naval 
expeditionary,”  “ship  board,”  and  “strike  carrier”)  can  good  candidates  to  engage  Action 
Plans  15  and  26. 


•  Action  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in  real  time. 
That  way  the  nearest  force  will  response  and  reduce  travel  distances. 

•  Action  26:  Expand  use  of  nuclear  power  in  the  fleet. 

Related  concepts  include  “multiple  hardware,”  “operating  time,”  and  “dashboard 
energy.” 
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The  matrix  for  Theme  905  that  the  PEs  involved  (“unmanned  systems,”  “surface 
ships,”  “nuclear  powered,”  “operational  environment,”  and  “water  treatment”)  can  be  good 
candidates  for  engaging  Action  Plans  18,  19,  20,  26,  31,  35,  4,  and  7. 


•  Action  Plan  18:  Offshore  basing. 

•  Action  Plan  19:  Implement  self-sustaining  support  infrastructure  on  all  Navy 
bases. 

•  Action  Plan  20:  Sails  on  vessels;  use  sails  that  are  foldable  on  the  sides  of 
vessels. 

•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 

•  Action  Plan  31 :  Add  “reducing  energy  consumption”  to  Battle  E  criteria. 

•  Action  Plan  35:  Create  3D/vertical  farms  for  use  in  growing  biofuels  and  crop 
for  human  consumption. 

•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 

•  Action  Plan  7:  Install  “sea  brakes”  that  generate  electricity,  like  a  Prius.  These 
could  be  used  to  aid  in  docking/slowing  ships  and  reduce  the  need  for  tugs. 
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The  match  matrix  for  Theme  132  shows  that  the  PEs  mentioned  (“additional  energy,” 
“ground  forces”  [e.g.,  PE  06021 31 M,  PE  0603640M;  PE  020631 3M;  PE  0602750N;  PE 
060501 3M;  PE  0604404N],  “harvesting  energy”  [e.g.,  PE  0602236N:  Warfighter 
Sustainment  Applied  Res;  PE  0603673N:  (U)Future  Naval  Capabilities  Advanced  Tech  Dev; 
PE  0601 153N:  Defense  Research  Sciences;  PE  0602123N:  Force  Protection  Applied  Res], 
“potential  energy,”  and  “hydrodynamic  forces”)  are  the  good  candidates  to  engage  Action 
Plans  14,  15,  17,  18,  34,  and  7. 

•  Action  Plan  14:  Recycle  everything  biological  into  fuel:  waste,  etc. 

•  Action  Plan  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in  real 
time.  That  way,  the  nearest  force  will  response  and  reduce  travel  distances. 

•  Action  Plan  17:  Energy  harvesting  satellites  in  outer  space  transmit  it  to  Earth 
via  microwave  or  laser  beam. 

•  Action  Plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore  bases 
in  critical  regions  such  as  the  South  China  Sea. 

•  Action  Plan  34:  Create  online  system  or  suggestion  card  system  for  Navy 
personnel  to  input  where  they  see  energy  savings  in  their  job. 

•  Action  Plan  7:  Install  “sea  brakes”  that  generate  electricity,  like  a  Prius.  These 
could  be  used  to  aid  in  docking/slowing  ships,  reduce  need  for  tugs. 
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The  match  matrix  for  Theme  787  suggests  that  the  PEs  (“energy  efficiency”  and  “fuel 
efficiency”)  can  be  viewed  as  “survivability  requirements”;  therefore,  any  PEs  related  to 
“survivability  requirements”  (e.g.,  PE  060321 6N:  Aviation  Survivability)  or  “operational 
requirements”  can  be  used  to  engage  Action  Plans  10,  11, 20,  27,  31,  34,  and  9. 

•  Action  Plan  9:  Composite  ship  design:  Explore  the  use  of  polymer  substrates 
for  improved  ship  structural  design. 

•  Action  Plan  1 0:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 


The  match  matrix  for  Theme  494  suggests  that  the  PEs  mentioned  (“shared 
information,”  “signal  intelligence,”  “share  data,”  “data  structures,”  “intelligence  systems,” 
“artificial  intelligence,”  and  “maritime  warfare”)  might  be  good  candidates  to  engage  Action 
Plans  16,  18,  26,  31,  and  36. 

•  Action  Plan  16:  Using  synthetic  lubricants  to  save  5%  to  25%  of  energy  costs. 

•  Action  Plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore  bases 
in  critical  regions  such  as  the  South  China  Sea. 

•  Action  Plan  36:  Become  more  efficient  at  structured,  logical  dialogue  to  find 
the  solutions  being  sought. 
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The  match  matrix  for  Theme  633  suggests  that  the  PEs  mentioned  (“advanced  tech” 
[e.g.,  PE  0603673N:  (U)Future  Naval  Capabilities  Advanced  Tech  Dev],  “greater  efficiency” 
[e.g.,  PE  0603747N:  Undersea  Warfare  Advanced  Tech],  and  “power  plants”)  can  be  good 
candidates  to  engage  Action  Plans  11,21,  and  4. 

•  Action  Plan  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet. 

•  Action  Plan  21 :  DoD  shore  facility  energy  independence:  Explore  use  of 
thorium-based  reactors  (liquid  fluoride  thorium  reactor  [LFTR])  for  power 
generation  off  the  grid. 


•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 


The  match  matrix  for  Theme  326  suggests  that  the  PEs  mentioned  (“energy 
security,”  “missile  defense,”  “operational  security,”  “cyber  security,”  “national  security,”  and 
“Naval  Postgraduate  School”)  might  be  good  candidates  to  engage  Action  Plans  17,  19,  4, 
27,  4,  35,  and  5. 

•  Action  Plan  17:  Energy  harvesting  satellites/space-based  solar  power. 

•  Action  Plan  19:  Implement  self-sustaining  support  infrastructure  on  all  Navy 
bases. 

•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 
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The  match  matrix  for  Theme  917  suggests  that  the  PEs  mentioned  (“nuclear  power,” 
“nuclear  technology,”  “safety  standards,”  “logistics  systems,”  “logistics  management,” 
“standards  development/data,”  and  “common  standards”)  might  be  good  candidates  to 
engage  Action  Plans  16,  18,  25,  26,  31,  34,  and  9. 


•  Action  Plan  34:  Create  online  system  or  suggestion  card  system  for  Navy 
personnel  to  input  where  they  see  energy  savings  in  their  job. 


The  match  matrix  for  Theme  579  suggests  that  the  PEs  mentioned  (“energy 
management,”  “composite  materials,”  “processing  capabilities,”  “supply  chains,”  “electrical 
energy,”  “hazardous  waste,”  “energy  absorbing,”  “sinks  heat,”  “heat  reduce,”  and  “naval 
academy”)  might  be  good  candidates  to  engage  Action  Plans  8,  20,  26,  and  9. 

•  Action  Plan  8:  Shore  energy  optimization  strategy:  Recommendations  for 
improvements  and  implementation. 


The  match  matrix  for  Theme  854  suggests  that  PEs  mentioned  (“turbine  engine,” 
“diesel  engine,”  “energy  sources,”  “power  sources,”  and  “greenhouse  gas”)  might  be  good 
candidates  to  engage  “behavior  modification”  related  Action  Plans  27,  8,  and  5. 

•  Action  27:  Upgrade  Navy  housing  with  SMART  grids  to  reduce  energy 
consumption.  By  individualizing  electricity/utility  bills  to  single  households, 
family  users  will  be  motivated  to  increase  energy  saving  efforts. 

•  Action  5:  Incentivize  behavior  to  reduce  electricity  usage  in  Navy  housing. 

•  Action  8:  Update  older  buildings  to  be  more  energy  efficient.  The  Navy  is  still 
using  buildings  that  are  almost  a  century  old. 

These  PEs  include,  for  example,  PE  0603573N:  Advanced  Surface  Machinery  Sys; 
PE  0603724N:  Navy  Energy  Program;  PE  0205633N:  Aviation  Improvements;  PE 
0206623M:  MC  Ground  Cmbt  Spt  Arms  Sys;  and  PE  0605864N:  Test  &  Evaluation  Support. 
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The  match  matrix  for  Theme  732  suggests  that  the  PEs  mentioned  (“ship  surface,” 
“fleet  surface,”  “power  management,”  “ship  power,”  “supplying  power,”  and  “generating 


“electric  warship,”  “training  centers”  and  “ocean  wave”).  These  PEs  include,  for  example,  the 
following: 

•  PE  0603563N:  Ship  Concept  Advanced  Design 

•  PE  0602123N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Cmbt  Services  Supt 

•  PE  06031 14N:  Power  Projection  Advanced  Technology 

•  PE  0601 153N:  Defense  Research  Sciences 


•  PE  06021 31 M:  Marine  Corps  Lndg  Force  Tech 
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The  match  matrix  for  Theme  449  suggests  that  the  PE  mentioned  (“power 
projection”)  can  be  used  to  engage  “social  media”  for  “fuel/energy  saving.” 
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•  Action  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet,  engage 
major  universities  to  create  a  cross-disciplinary  curriculum  for  “energy  design” 
in  all  fields  for  all  forms  of  energy. 
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The  match  matrix  for  Theme  682  suggests  that  the  PEs  mentioned  (“ship 
construction,”  “ship  operations,”  “fleet  operations,”  “military  construction,”  and  “operations 
research”)  can  be  good  candidates  to  engage  Action  Plans  10,  26,  and  6. 


•  Action  Plan  1 0:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 

•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 


•  Action  Plan  6:  Implement  large  umbrellas  for  ships  to  use  shading  to  keep 
ship  cooler;  also  use  “carport”  structures  for  ships  docked  on  the  pier. 
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The  match  matrix  for  Theme  257  suggests  that  the  PEs  mentioned  (“parts 
replacement,”  “communication  equipment,”  “air  wing,”  “communication  data,”  and  “urban 
environments”)  might  be  good  candidates  for  Action  Plans  16,  18,  27,  28,  34,  and  35. 


•  Action  16:  Using  synthetic  lubricants  to  save  5%  to  25%  of  energy  costs. 

•  Action  18:  Offshore  basing. 

•  Action  27:  Upgrade  Navy  housing  with  SMART  grids  to  reduce  energy 
consumption.  By  individualizing  electricity/utility  bills  to  single  households, 
family  users  will  be  motivated  to  increase  energy  saving  efforts. 

•  Action  28:  Power  on-board  minor  electronics  with  stationary  bikes  used  for 
personnel  fitness  training. 

•  Action  34:  Online  feedback  and  social  networking. 

•  Action  35:  3D  farming:  Less  land  use  and  local  agriculture  reducing  fuel  use 
and  potential  location  of  bio-fuel  crops. 
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The  match  matrix  for  Theme  198  suggests  that  the  PEs  mentioned  (“energy  saving,” 
“fuel  savings,”  “cost  savings,”  “fuel  cell,”  “cell  technologies,”  “storage  energy,”  and  “storage 
systems”)  might  be  good  candidates  to  engage  Action  Plans  related  to  these  concepts. 


The  resulted  matrices  from  this  task  will  help  design  the  specific  questions  to 
address  the  issues  on  a  program-to-program  basis  to  continue  the  energyMMOWGLI  game 
with  acquisition  professionals  in  the  acquisition  research  community  in  the  future. 
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